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DIRECT  COMPARISON  OF  INTUITIVE,  QUASI-RATIONAL 
AND  ANALYTICAL  COGNITION 


Comparison  of  the  efficacy  of  intuitive  and  analytical  cogniffon'Tias  been 
a  major  topic  in  judgment  and  decision  research  since  its  inception  (for  a 
review  of  approaches  to  this  topic,  see  Hammond,  McClelland  &  Mumpower,  1980). 
Although  intuitive  cognition  is  largely  believed  to  be  inaccurate, 
systematically  biased,  and  to  produce  judgments  in  which  persons  are 
overconfident,  (for  reviews  see  Slovic,  Fischhoff  &  Lichtenstein,  1977; 
Einhorn  &  Hogarth,  1981),  Hastie  in  his  review  (1983)  concludes  that:  "none 
of  the  applications  [of  probability  theory]  to  social  judgment  provide 
compelling  arguments  that  human  reasoning  deviates  from  normative  model 
prescriptions"  (p.  511).  Our  premise  is  that  both  conclusions  are  based  on 
research  that  is  restricted  in  the  following  ways: 

1.  Comparisons  are  indirect. 

The  most  widely  used  research  method  compares  the  intuitively  derived 
judgments  of  subjects  with  analytically  derived  answers  generated  by 
formal  models  such  as  Bayes1  Theorem  and  multiple  regression  equations 
(see,  for  example,  Kahneman,  Slovic,  &  Tversky,  1982;  Hammond,  Stewart, 
Brehmer,  &  Steimnann,  1975).  Informative  as  these  comparisons  may  be, 
they  do  not  directly  compare  the  achievement  of  intuitive  and  analytical 
cognition  undertaken  by  the  same  person.  Direct  comparisons  are  needed  in 
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order  to  know  which  mode  of  human  cognition  Is  superior.  (For  a  general 
exposition  regarding  these  comparisons  see  Tweney,  Doherty  &  Mynert, 
1981.) 

2.  Research  subjects  lack  analytical  competence. 

Research  subjects — typically  college  sophomores--generally  lack 
knowledge  of  any  analytical  principles  that  would  enable  them  to  engage  in 
analytical  cognition.  Moreover,  they  are  seldom  given  the  time  or 
opportunity  to  do  so.  Since  the  subject's  analytical  competence  is  not 
examined  in  parallel  with  his/her  intuitive  competence,  the  substitution 
of  formal  models  to  represent  analytical  cognition  constitutes  a  special 
comparison.  That  is,  intuitive  cognition  is  set  in  competition  with  a 
form  of  analytical  cognition  refined  by  centuries  of  study  by  thousands  of 
individuals  over  their  lifetimes.  Although  such  comparisons  are 
undeniably  important,  they  do  not  exhaust  our  range  of  interest  in 
cognition;  they  do  not  inform  us  about  what  happens  when  the  same  person 
engages  in  different  forms  of  cognitive  activity.  They  cannot,  for 
example,  inform  us  about  the  possible  disadvantages  as  well  as  advantages 
of  engaging  in  analytical  cognition. 

3 .  Intuition  and  analysi s  remain  obscure  concepts . 

Researchers  in  cognition  almost  never  explain  what  they  mean  by 
intuition,  although  they  take  great  pains  to  differentiate  precisely  among 
formal,  analytical  models  of  cognition.  As  a  result  it  is  customary  to 
explain  that  intuition  is  what  analysis  is  not.  Brooks  (1978),  for 
example,  compares  "analytic  and  non-analytic  concept  formation."  Kahneman 
and  Tversky  (1982)  indicate  that  "a  judgment  is  cabled  intuitive  if  it  is 
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reached  by  an  informal  and  unstructured  mode  of  reasoning,  without  the  use 
of  analytical  methods  or  deliberate  calculations"  (p.  494).  Lven 
philosophers  (e.g.,  Cohen,  1981)  who  criticize  psychological  research 
regarding  intuitive  judgments  of  probability  fail  to  say  what  they  mean  by 
this  term.  As  a  result,  the  task  properties  that  induce  either  form  of 
cognition  are  not  specified,  thus  making  their  comparison  difficult,  if 
not  impossible. 

4.  Methodological  and  substanti ve  cognition  are  not  differentiated. 

Successful  cognitive  efforts  require  substantive  as  well  as 
methodological  competence;  neither  alone  can  insure  empirical  success 
(cf. ,  Hammond,  1966,  pp.  68-75;  Wason  &  Johnson-Laird,  1972;  Adelmen, 
1981).  Topical  knowledge  can  be  as  important  as  the  knowledge  of 
methodological  principles.  These  two  aspects  of  cognition  are  generally 
studied  separately,  however.  Judgment  and  decision  researchers  contrast 
their  subjects'  intuitive  methodological  cognition  (efforts  to  combine 
information)  with  analytically  deriyed  normative  methodological  rules  for 
combining  information  (e.g.,  Bayes1  Theorem),  irrespective  of  subjects’ 
topical  knowledge  (see  L inhorn  &  Hogarth,  1981;  Slovic  et  al.,  1977). 
Researchers  who  study  proolem-sol ving,  on  the  other  hand,  generally 
contrast  their  subjects'  manipulation  of  the  substantive  materials  of  the 
task  with  scientific  truths  or  experts'  behavior  (see,  for  example, 
Larkin,  McDermott,  Simon  &  Simon,  1980;  also  Duda  &  Shortliffe,  1983) 
without  regard  for  subjects'  methodological  competence.  If  the  efficacy 
of  intuitive  and  analytical  cognition  are  to  be  directly  compared, 
however,  these  two  aspects  of  cognition  need  to  be  examined  together,  for 
they  can  be  compensatory  in  many  task  circumstances.  (An  explicit  effort 
to  study  both  can  be  seen  in  Fox,  1980.) 
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5 .  Evaluations  of  Intuitive  methodological  cognition  are  norm-contingent. 


Considerable  uncertainty  exists  about  which  normative  rules  should  be 
used  to  evaluate  intuitive  methodological  cognition.  The  sharply 
divergent  views  of  philosophers  on  this  topic  (see,  for  example,  Kyburg  & 
Smokier,  1980,  Kyburg,  1974,  and  Levi,  1980)  are  reflected  in  the 
responses  to  Cohen's  (1981)  criticism  of  judgment  and  decision  research 
that  uses  the  standard  probability  calculus  to  evaluate  intuitive 
methodological  cognition.  The  number  and  variety  of  normative  inductive 
rules  cited  in  these  responses  as  the  proper  rule  to  follow  are  clear 
evidence  that  both  frequentists  and  subjectivists  have  failed  to  solve  the 
fundamental  problem  of  induction.  Until  they  succeed,  the  research 
psychologist  who  attempts  an  empirical  evaluation  of  the  extent  to  which 
intuitive  cognition  conforms  to  normative  rules  will  inevitably  produce 
results  contingent  upon  the  normative  rules  s/he  prefers--as  those  who 
prefer  different  rules  will  be  quick  to  point  out.  Einhorn  and  Hogarth 
(1981)  cite  other  difficulties  with  normative  rules  in  their  review  when 
they  note  that  choice  of  an  optimal  model  is  "conditional  on  certain 
environmental  assumptions  and  a  specific  time  horizon"  (p.  55)  and 
conclude  their  discussion  by  observing  that  "To  consider  human  judgment  as 
suboptimal  without  discussion  of  the  limitations  of  optimal  models  is 
naive"  (p.  56). 


Evaluations  of  substantive  analytical  cognition  are  less  vulnerable 
to  norm-contingent  restrictiqns,  however,  because  a  subject's  production 
of  a  substantive  law  or  rule  can  be  evaluated  with  respect  to  well  agreed 
upon  empirical  events  (cf.  Scribner,  1977,  who  distinguishes  between 
"theoretical"  and  "empirical"  standards  for  evaluating  logical  processes). 
Therefore,  empirical  achievement  is  a  criterion  for  cognitive  activity 
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that  deserves  as  much  or  more  interest  as  conformity  with  one,  among  many, 
normative  rules.  (See  Hammond  et  al . ,  1980,  on  "logical  vs.  empirical 
optimality,"  p.  215ff;  see  also  Einhorn  &  Hogarth,  1982:,  on  "truth  vs. 
accuracy ) 

6.  Evaluations  of  intuiti ve  methodological  cognition  are  task  contingent. 

Current  conclusions  about  intuitive  competence  in  human  beings  are 
largely  restricted  to  the  performance  of  subjects  working  with 
methodological  problems  that  have  been  constructed  to  be  readily  treated 
by  the  methods  of  the  subjective  interpretation  of  the  standard 
probability  calculus.  But,  as  noted  above,  cognitive  problems  involve 
substantive  competence  as  well  as  methodological  competence.  Therefore, 
comparison  of  the  efficacy  of  intuition  and  analysis  requires  examination 
in  tasks  that  offer  challenges  to  empirical  accuracy  as  well  as  intuitive 
methodological  competence. 

Examination  of  performance  in  substantive  tasks  representative  of  a 
person's  intellectual  environment  should  be  at  least  as  instructive  as  an 
examination  of  performance  in  task  conditions  specifically  constructed  to 
conform  to  normative  methodological  rules.  Johpson-Lai rd  a, id  Wason,  for 
example,  note  in  their  1977  "Postscript":  "by  attempting  to  relate 
the... task  more  closely  to  the  subjects'  experience,  performance  was 
dramatically  improved"  (1977,  p.  151).  Einhorn  and  Hogarth  (1981) 
conclude:  "It  is  essential  to  emphasize  that  the  cognitive  approach  has 
been  concerned  primarily  with  how  tasks  are  represented.  The  is^ue  of  why 
tasks  are  represented  in  particular  ways  has  not  yet  been  addressed"  (p. 
57).  Indeed,  the  general  absence  of  the  description  and  classification  of 
cognitive  tasks  has  long  been  emphasized  as  a  serious  shortcoming  in 
judgment  and  decision  research  (see  Hammond,  1954;  Edwards,  1971;  Slovic 
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&  Lichtenstein,  1971;  tinhorn  &  Hogarth,  1981)  and  continues  to  be 

(Hastie,  1983),  but  without  tangible  result. 

In  sum.  In  addition  to  making  indirect  comparisons  it  is  necessary  to 
directly  compare  an  analytically  competent  person's  use  of  intuitive  and 
analytical  cognition,  to  permit  the  subject  to  employ  either  or  both  in 
relation  to  substantive  as  well  as  methodological  cognition,  and  to  evaluate 
his  or  her  achievement  empirically.  In  the  present  study,  therefore,  we  do 
the  following: 

1.  directly  compare  intuitive  and  analytical  cognition  in  the  same 
person; 

2.  use  as  subjects  analytically  competent  persons  so  that  both  Intuitive 
and  analytical  cognition  can  be  brought  to  bear  on  the  same  task; 

3.  specify  the  properties  of  intuitive  and  analytical  cognition  and  the 
properties  of  tasks  that  evoke  each; 

4.  include  substantive  as  well  as  methodological  aspects  of  cognition; 

5.  contrast  the  efficacy  of  intuition  and  analysis  in  terms  of  empirical 
achievement;  and 

6.  use  problems  representati ve  of  the  subject's  intellectual  environment. 
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Theoretical  background 

A  Cogni ti ve  Continuum  vs.  a  Cognitive  Dichotomy 

Our  basic  premise  is  that  cognitive  activity  is  not  a  dichotomy  of 
intuition  and  analysis  but  rather  a  continuum  marked  by  intuition  at  one  pole 
and  analysis  at  the  other.  Unlike  the  traditional  dichotomous  premise,  a 
cognitive  continuum  permits  a  quasi-rational  compromise  between  intuitive  and 
analytical  cognition;  thus  a  form  of  cognition  more  common  than  either  pure 
intuition  or  analysis  may  be  described.  (For  previous  use  of  the  concepts  of 
cognitive  continuum  and  cognitive  compromise  see  Brunswik,  1952,  1956; 

Hammond,  1955;  Hammond  &  Brehmer,  1973;  Hammond  et  al . ,  1975;  Hammond  et 
al . ,  1980;  Hammond,  Note  1;  see  also  Anderson,  Deane,  Hammond,  McClelland  & 
Shanteau,  1981.) 

Modes  of  Cognition 

Intuition  and  analysis  can  be  distinguished  by  the  relative  degree  of: 
(a)  cognitive  control  (in  intuition,  low;  in  analysis,  high);  (b)  rate  of 
data  processing  (in  intuition,  rapid,  i.e.,  as  brief  as  microseconds;  in 
analysis,  slow,  i.e.,  as  long  as  hours);  (c)  conscious  awareness  of  process 
(in  intuition,  low;  in  analysis,  high);  (d)  type  of  organizing  principle  (in 
intuition,  a  weighted  average;  in  analysis,  other,  task-specific  principles); 
(e)  type  of  error  (in  intuition,  normally  distributed;  in  analysis,  few,  but 
large  errors);  (f)  type  of  confidence  (in  intuition,  confidence  in  answer  but 
not  method;  in  analysis,  confidence  in  method,  not  answer).  (For  further 
distinctions,  see  Hammond,  Note  1.)  The  compromise  form  of  cognition, 
quasi-rationality  or  "common  sense,”  includes  properties  from  both  types  of 
cognition  and,  therefore,  is  described  in  terms  of  the  number  and  nature  of 
the  cognitive  properties  it  includes  from  both. 
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Task  Properties  That  Induce  Different  Modes  of  Cognition 

If,  as  in  the  present  study,  subjects  are  not  provided  with  feedback ,  the 
task  properties  that  differentially  induce  intuition  and  analysis  include: 

(a)  number  of  cues  available  (in  intuition,  large  [>  5];  in  analysis,  small); 

(b)  the  order  i n  which  cues  are  displayed  (in  intuition,  simultaneous;  in 
analysis,  sequential);  (c)  the  type  of  cue  measurement  required  (in 
intuition,  perceptual;  in  analysis,  objective,  as  with  instruments);  (d)  cue 
distribution  characteristics  (in  intuition,  continuous,  highly  variable, 
normally  distributed;  in  analysis  dichotomous,  valued  in  terms  of  specific 
numbers,  distributions  unknown);  and  (e)  redundancy  among  cues  (in  intuition, 
high;  in  analysis,  low).  (See  Hammond,  Note  1,  for  further  elaboration.) 
Quasi-rationality  is  induced  to  the  extent  that  tasks  contain  properties  from 
both  types  of  polar  task  conditions.  We  do  not  claim  that  all  of  these 
properties  must  be  present  in  order  to  locate  a  task  at  either  pole  of  the 
continuum,  nor  do  we  know  their  relative  importance  or  their  interactive 
effects  that  produce  quasi  rationality.  We  assert  only  that  specification  of 
these  task  properties  is  a  useful  guide  for  direct  comparison  of  the  relative 
efficacy  of  different  modes  of  cognition. 

Objectives 

Four  aspects  of  the  cognitive  continuum  theory  are  examined  in  the 
context  of  the  six  conditions  indicated  above.  Because  of  the  long-standing 
interest  in  the  relative  efficacy  of  modes  of  cognition  we  first  examine 
differences  in  the  empirical  accuracy  of  highway  engineers'  judgments  of 
safety  in  the  intuitive,  quasi -rational  and  analytical  modes.  Highway 
engineers  were  chosen  as  subjects  because  of  their  frequent  professional  use 
of  all  three  modes  of  cognition.  Judgments  of  highway  safety  were  chosen 
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because,  as  we  show  below,  the  properties  of  this  task  should  induce  quasi 
rationality,  thus  illustrating  the  importance  of  the  middle  range  of  the 
cognitive  continuum.  (An  intuition-inducing  task--judgi ng  highway 
aesthetics--and  an  analysis-inducing  task-- judgi ng  highway  capaci ty--were  also 
employed,  but  space  prevents  their  discussion  here.) 

Second,  because  of  the  important  theoretical  role  of  cognitive 
control-predicted  to  be  low  in  intuition,  middle-level  in  quasi  rationality, 
and  high  in  analysis--we  examine  differences  in  cognitive  control  in  each  of 
the  three  modes.  More  specifically,  the  relative  contributions  of  knowledge 
and  cogniti ve  control  to  accuracy  are  compared.  Thec°  two  aspects  of 
cognition,  identified  by  Hammond  and  Summers  (1972),  have  been  found  to  be 
empirically  significant  in  numerous  studies  of  multiple-cue  probability 
learning  (see,  e.g.,  Brehmer,  1979),  of  interpersonal  conflict  and 
interpersonal  learning  (see  Brehmer,  1972,  1976;  Holzworth,  in  press; 
Brehmer  &  Hammond,  1977  for  a  review),  of  the  differential  effects  of  various 
psychoactive  drugs  on  interpersonal  learning  and  interpersonal  conflict  among 
psychotic  patients  (Gillis,  1975,  1978),  and  in  clinical  judgment  (e.g., 
Fisch,  Joyce,  Hammond  &  O'Reilly,  1982;  Kirwan,  Chaput  de  Saintonge,  Joyce,  & 
Currey,  in  press).  In  this  study  knowledge  and  cognitive  control  are  examined 
to  determine  whether  they  provide  different  contributions  to  achievement  under 
the  three  modes  of  cognition  within  the  same  person. 

One  of  the  advantages  of  the  direct  comparison  of  human  intuitive  and 
analytical  cognition  is  that  the  errors  made  in  the  analytical  mode  of 
cognition  can  be  observed  and  compared  with  the  errors  of  intuition.  This 
comparison  cannot  be  made  when  intuitive  cognition  is  compared  only  with  a 
formal  model.  For  in  that  case  the  answer  provided  by  the  formal  model 
exhausts  the  concept  of  truth;  by  definition,  errors  do  not  exist.  Therefore 
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our  third  objective  Is  to  compare  the  errors  produced  by  analytical  as  well  as 
Intuitive  and  quasi -rational  cognition. 

Brunswik  has  already  argued  and  demonstrated  (1956,  pp.  09-93)  that 
intuitive  cognition  produces  errors  normally  distributed  around  the  correct 
answer  because  "intuitive  perception  must  integrate  many  avenues  of  approach, 
or  cues. ..none  of  which  is  foolproof  or  fully  ecologically  valid"  (p.  92). 
Because  analytical  cognition  proceeds  in  the  opposite  fashion,  it  produces 
many  exactly  correct  answers,  but  its  errors  are  likely  to  be  extreme.  We 
therefore  ask  whether  intuitive  cognition  produces  a  normal  distribution  of 
errors  centered  on  the  correct  answer;  whether  analytical  cognition  produces 
precisely  correct  answers  together  with  highly  incorrect  answers;  and  whether 
quasi-rationality  produces  a  distribution  of  errors  lying  between  those 
generated  by  the  polar  modes  of  cognition. 

Fourth,  the  relation  between  confidence  and  performance  is  examined  to 
discover  whether  confidence  matches  performance  (Oskamp,  1965;  Lichtenstein, 
Fischhoff  &  Phillips,  1982):  is  an  engineer  most  confident  in  that  cognitive 
mode  in  which  he  performed  best  and  least  confident  in  the  mode  under  which  he 
performed  most  poorly?  The  theory  outlined  above  suggests  an  additional 
hypothesis.  Since  the  method  of  intuitive  cognition  produces  rapid, 
nonretraceable  answers  and  since  these  answers  are  based  on  multiple  "avenues 
of  approach  or  cues,"  subjects  should  be  less  confident  in  the  intuitive 
method  than  in  intuitive  answers.  The  opposite  should  be  true  in  the 
analytical  mode,  in  which  the  subject's  attention  is  focused  on  the 
organization  of  cues. 
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Method 


Subjects 

Twenty-one  male  highway  engineers,  30-70  years  of  age,  served  as  research 
subjects.  Since  engineers  are  professionally  trained  to  cope  with  problems 
that  have  substantive  analytical  components,  the  intuitive  and  analytical 
cognitive  efforts  of  the  same  subject  may  be  directly  compared.  The 
engineers'  task  was  to  evaluate  the  safety  of  highways,  a  complex  problem 
representative  of  those  encountered  in  their  work. 


Independent  Variables 


In  the  intuition-inducing  condition  film  strips  of  one-  to  three-mile 
segments  of  forty  two-lane  rural  Colorado  highways  were  presented.  Engineers 
judged  the  safety  of  each  segment  solely  on  the  basis  of  the  visual  material 
in  the  film  strips  (see  Figure  1).  This  form  of  presentation  meets  the 
conditions  for  inducing  intuitive  cognition  by  requiring  the  engineers  to 
observe  a  large  number  of  cues  contemporaneously  displayed,  and  to  measure 
them  by  unaided  visual  perception.  The  values  of  the  cues  are  generally 
continuous  and  normally  distributed;  the  cues  are  frequently  redundant.  The 
engineers  were  neither  asked  nor  given  the  opportunity  to  organize  the  task 
materials  explicitly. 
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Figure  2.  Bar-graph  Profile  of  Highway  #27 
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normal  distributions.  The  engineers  were  not  given  the  opportunity  to 
organize  the  task  materials  explicitly  and  were  not  asked  to  explicate  or 
justify  their  judgments.  On  the  other  hand,  the  number  of  observable  cues  is 
reduced  to  ten.  The  cue  values  are  expressed  as  numbers,  and  thus  the 
engineers  are  not  required  to  measure  them  perceptually.  The  bar-graph 
profiles  also  permit  easy  cross-comparison  of  the  values  of  various  cues. 

In  the  analysis-inducing  condition  each  engineer  was  requested  to 
construct  and  justify  a  mathematical  equation  for  predicting  safety  (see 
Appendix  A  for  examples).  The  subject  was  instructed  to  think;  pencil,  paper 
and  hand  calculators  were  made  available;  the  engineer  was  told  that  a 
certain  amount  of  time  would  be  allowed  for  the  completion  of  the  formula,  but 
that  the  time  period  would  be  extended  until  he  completed  the  task.  Each 
engineer  was  assigned  to  one  of  three  subgroups  within  the  analytical 
condition.  In  the  "minimal  guidance"  subgroup  (n  =  12)  engineers  were  allowed 
to  work  through  the  task  with  a  minimum  of  instructions  and  requests  from  the 
researchers.  Six  engineers  in  this  group  were  encouraged  to  complete  their 
formulas  in  twenty  to  thirty  minutes;  the  other  six  were  give  a  target  of 
forty-five  minutes.  In  the  "think  aloud"  subgroup  (n  =  6)  engineers  were 
requested  to  think  aloud  as  they  constructed  their  equations  so  that  their 
analytical  efforts  could  be  monitored  during  the  task.  In  the  "maximal 
guidance"  subgroup  (n  -  3)  engineers  constructed  their  equations  according  to 
a  detailed  written  procedure  (see  Appendix  B)  designed  to  increase  the 
likelihood  of  a  systematic  approach  to  the  problem.  (Samples  of  engineer's 
remarks  regarding  their  cognitive  activity  in  each  mode  are  presented  in 
Appendix  C. ) 
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These  three  sub-conditions  were  used  because  they  offered  a  variety  of 
information.  The  minimal  guidance  condition  revealed  how  engineers  would  go 
about  constructing  a  formula  on  their  own.  The  "think  aloud"  condition 
provided  an  understanding  of  the  engineers'  process  of  constructing  an 
equation  which  would  not  otherwise  have  been  gained  (see  Appendix  D).  The 
maximal  guidance  condition  showed  whether  a  structural  approach  was  feasible. 

It  was  not  our  purpose  to  determine  the  effects  of  the  different 
sub-conditions.  As  would  be  expected,  however,  engineers  in  maximal  guidance 
and  engineers  who  were  given  a  45-minute  target  in  minimal  guidance  took  more 
time  than  the  others.  There  were  no  significant  differences  in  achievement 
among  the  subgroups. 

Dependent  Variables 

The  effect  of  the  intuitive,  quasi -rational ,  and  analytical  conditions 
were  examined  with  respect  to  (a)  each  engineer's  degree  of  achievement  in 
predicting  safety  accurately,  (b )  the  differential  contributions  of  knowledge 
and  cognitive  control  to  this  achievement,  (c)  the  relative  frequency  of 
different  types  of  errors  made  in  each  of  the  three  modes  of  cognition,  and 
(d)  the  relation  of  confidence  to  performance  in  each  mode. 

Achievement  was  measured  by  the  correlation  between  an  engineer's 
estimate  of  safety  and  the  accident  rate  of  each  highway  segment  (the 
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Knowledge  and  cognltl ve  control  were  derived  from  the  parameters  of  the 
Lens  Model  Equation  (Hammond  et  al.,  1975 ) : 

r  6  Ra  RP  +  C  J  1  -  R  2  J  1  -  r7 

a  e  s  e  x  s 

where 

r  =  achievement,  the  correlation  between  the  engineer's 

a 

judgments  and  criterion  values 
G  =  the  correlation  between  judgments  and  criterion  values 
corrected  for  attenuation  due  to  less  than  perfect 
linear  predictability  in  each 
Re  =  environmental  predictability  (linear  form) 

R$  =  subject's  predictabil ity  (linear  form) 

C  =  correlation  between  residuals  from  linear  predictions 
of  criterion  and  residuals  from  linear  predictions  of 
subject's  judgments. 

In  the  absence  of  significant  correlations  between  residuals  (trivial 

values  of  C  in  the  above  equation)  then  r  =  G  R  R  .  Under  these  conditions 

G  represents  the  engineer's  knowledge  because  it  indicates  what  the  subject's 
achievement  would  have  been  if  he  had  executed  his  judgment  policy  with 

perfect  cognitive  control  (i.e.,  Rg  =  1.00)  and  if  the  environmental  task 

criterion  were  perfectly  predictable  from  the  cues  (i.e.,  Rg  =  1.00). 

Cogniti ve  control  is  appropriately  measured  by  R$  in  the  equation  since 
there  was  little  evidence  in  this  study  of  lack  of  fit  of  the  linear  model. 
(See  Hammond  et  al . ,  1975,  for  a  detailed  discussion  of  the  distinction 
between  consistency  and  cogniti ve  control . ) 
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Pi fferential  errors  produced  by  each  engineer  are  evaluated  by  (a) 
examination  of  the  distribution  of  deviations  of  judgments  around  the  correct 
answer,  and  (b)  examination  of  mistakes  made  in  the  process  of  formulating  the 
equation  and  evaluation  of  their  consequences  for  accurate  prediction. 

Confidence  in  answers  in  the  intuitive  and  quasi-rational  modes  is 
measured  by  the  mean  of  the  engineer's  confidence,  on  a  1-10  scale,  in  each  of 
his  judgments.  Confidence  in  answers  in  the  analytical  mode  and  confidence  in 
method  for  all  three  modes  was  measured  by  questions  at  the  conclusion  of  the 
session. 

Procedure 

Statistical  Properties  of  the  Task 

The  same  set  of  forty  highway  segments  was  used  in  all  conditions.  The 
criterion  for  the  accuracy  of  judgments  is  the  accident  rate,  averaged  over 
seven  years,  for  each  highway  segment.  Accident  rate  is  defined  as  the  total 
number  of  accidents  (involving  fatalities,  injuries,  or  property  damage  only) 
divided  by  the  number  of  vehicle  miles  traveled.  Due  to  its  extremely  high 
accident  rate,  one  highway  was  dropped  from  the  analysis. 

Highways  were  measured  on  ten  dimensions,  chosen  for  inclusion  in  the 
study  on  the  basis  of  discussions  with  highway  safety  experts  who  indicated 
the  information  they  considered  essential  for  evaluating  the  safety  of  a  road 
(see  Table  1  for  list).  Eight  of  these  measures  were  available  from  highway 
department  records;  two  measures  (number  of  curves  per  mile  and  number  of 
obstacles  per  mile)  had  to  be  counted  by  the  experimenters  from  visual 
inspection  of  film  strips  of  each  highway  segment.  The  beta  weights  for  each 
dimension  or  cue  in  predicting  accident  rate  are  also  presented  in  Table  1. 
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Visual  examination  of  the  scatter-plots  of  the  relations  between  each  cue  and 
the  criterion  indicated  little  if  any  nonlinear  co-variation.  This  finding 
was  supported  by  the  results  from  calculation  of  the  contribution  of  squared 
terms  and  interactions  to  accident  rates. 

Table  1 

Highway  Characteristics  (cues)  Related  to  Accident  Rate 
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An  optimally  weighted  linear  multiple  regression  model  of  the  task 
indicates  that  Rg  =  .b63,  corrected  for  estimated  shrinkage  =  .809. 
Application  of  equal  weights  and  linear  functions,  each  cue  given  the  sign 
that  appeared  in  the  best  fit  equation,  yields  an  of  .769  (corrected  for 
shrinkage  =  .667).  The  intercorrelations  among  the  ten  cues  and  the  criterion 
are  presented  in  Table  2. 

Rating  Scales 

The  judgment  scale  for  each  task  condition  was  appropriate  to  the 
cognitive  activity  induced.  An  abstract  rating  scale  from  1  (safe)  to  10 
(unsafe)  was  used  for  the  film  strips  to  induce  intuitive  cognition.  In  the 
bar-graph  presentation  and  in  the  task  requiring  the  construction  of  a 
formula,  a  scale  from  0  to  32  accidents  per  million  vehicle  miles  traveled  was 
employed  because  this  specificity  is  compatible  with  calculation  and  thus  with 
analytical  cognition.  Transformations  to  a  common  scale  were  made  for 
purposes  of  data  analysis  and  are  described  below. 

T  rial  s 

All  engineers  were  presented  with  the  tasks  in  the  same  order:  first., 
the  film  strips;  second,  the  bar  graphs;  third,  the  materials  for  formula 
construction.  It  was  not  appropriate  to  counterbalance  the  order  of 
presentation  because  analytical  work  requiring  use  of  certain  cues  in  an 
explicit  fashion  would  have  strongly  influenced  subsequent  intuitive 
judgments,  whereas  the  reverse  is  not  true  (see  Jones  and  Harris,  1982). 

In  the  intuition-inducing  mode,  ten  of  the  forty  highways  were  shown 
twice;  in  the  quasi  rationality-inducing  mode  sixteen  highways  were  shown 
twice.  These  repetitions  permitted  calculation  of  repeated  trials  reliability 
for  each  engineer. 


Intercorrelations  among  Cues 
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T  ime 


Response  times  were  recorded  in  all  three  presentation  modes,  but  are 
artifacts  of  the  task  conditions  rather  than  descriptions  of  the  subjects' 
behavior;  that  is,  presentation  of  a  film  strip  required  more  time  than 
presentation  of  a  bar  graph.  The  mean  response  time  in  the  intuitive  mode  was 
78  seconds  and  in  the  quasi -rational  mode,  22  seconds. 

In  the  analytical  mode  the  response  time  varied  within  the  three 
subgroups.  The  eighteen  engineers  in  the  think  aloud  and  minimal  guidance 
subgroups  were  encouraged  to  complete  their  formulas  in  twenty  to  forty-five 
minutes;  but  seven  of  them  took  at  least  an  hour  at  this  task  (maximum  =  115 
minutes).  No  time  constraints  were  imposed  on  the  engineers  in  the  maximal 
guidance  condition;  response  times  ranged  from  138  to  250  minutes  with  a  mean 
of  180  minutes. 


RESULTS 


Relative  Efficacy  of  Three  Modes  of  Cognition 


Achievement 

Individual  differences  in  achievement.  Achievement  (r  )  is  measured  by 

- — ■  -  ■  -  ■  - -  - - - - — -  a 

the  correlation  between  an  engineer's  judgments  about  the  safety  of  each  of 
the  39  highways  and  the  accident  rates  of  those  highways.  Data  from  repeated 
judgments  in  the  intuitive  and  quasi-rational  modes  were  not  included.  For 
the  analytical  condition,  the  engineer's  formula  was  applied  to  each  of  the  39 
highways,  and  the  answers  thus  produced  were  correlated  with  the  accident 


rates. 
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Wide  individual  differences  occurred  in  all  modes  of  cognition  (Table  3). 
In  the  intuitive  mode,  the  engineer's  achievement  correlations  ranged  from 
.071  to  .636,  with  a  median  of  .576.  In  the  quasi-rational  mode,  r  ranged 

a 

between  .000  and  .738,  with  a  median  of  .516.  In  the  analytical  mode,  ra's 
ranged  from  -.226  to  .731,  with  a  median  of  .467.  Mean  interjudge  agreement 
in  the  intuitive  mode  is  .58;  in  the  quasi-rational  mode,  .47;  and  in  the 
analytical  mode,  .17.  The  correlation  between  engineers'  achievement  in  the 


intuitive 

and 

quasi-rational  modes  is 

.344; 

between  achievement  in 

the 

intuitive 

and 

analytical  modes,  .457; 

and 

between  achievement 

in 

the 

quasi-rational 

and  analytical  modes,  .U32 

(Table 

4). 

Since 

the 

engineers'  general  policies 

for  judging  highway  safety 

i  n 

the 

intuitive 

and 

quasi-rational  modes  were  all 

modeled  by  best-fit 

multiple 

regression  analysis,  individual  differences  in  achievement  in  these  modes 
cannot  be  attributed  to  the  structure  of  this  single  model.  In  the  analytical 
mode,  however,  judgment  policies  were  not  modeled  but  instead  directly 
expressed  by  the  engineers  as  formulas.  Individual  differences  in  achievement 
in  the  analytical  mode  could  have  been  caused,  in  part,  by  the  wide  variety  in 
the  structure  of  those  formulas  (see  Appendix  A).  It  turned  out,  however, 
that  the  various  structural  features  of  the  formulas  are  not  related  to 
achievement. 

In  the  intuitive  mode,  the  more  experience  the  engineer  had  (measured  by 
age,  years  of  work,  or  years  of  education),  the  lower  his  cognitive  control, 
R$;  otherwise  there  were  no  significant  relations  between  experience  and  the 
various  measures  of  achievement,  knowledge  and  consistency.  The  existence  of 
wide  individual  differences  in  achievement  among  the  engineers  in  all  three 
cognitive  inodes  thus  supports  the  decision  to  study  separately  the  performance 


Table  3 

Lens  Model  Equation  Components  and  Repeated  Trials  Reliabilities 
for  Engineers  in  Each  Condition 
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of  each  engineer  (cf.  Brunswlk,  1952;  Luce,  1959;  Tversky,  1972;  Shanteau, 
1975;  Hammond  et  al . ,  I960,  pp.  115-127;  Hammond  &  Wascoe,  1980;  Hammond, 
Note  1). 

Achievement  in  three  modes  of  cognition.  In  order  to  make  direct 
comparisons  of  each  subject's  performance  in  each  mode  of  cognition,  the 
efficacy  of  each  mode  for  each  engineer  was  evaluated.  The  basic  analysis  of 
data  thus  consisted  of  rank  ordering  each  engineer's  performance  across  modes. 
Means  are  provided  for  general  information  but  are  not  included  in  the 
statistical  tests. 

A  chi-square  test  of  the  hypothesis  that  one  mode  of  cognition  was 
superior  to  the  others  resulted  in  a  failure  to  reject  the  null  hypothesis 
(see  Table  5).  There  were,  however,  only  four  engineers  whose  achievement  was 
highest  in  the  intuitive  mode.  Comparison  of  the  six  orders  of  achievement 
did  result  in  rejection  of  the  hypothesis  of  a  chance  distribution  (p  <  .01). 
The  data  suggest  that  all  nine  engineers  who  performed  best  in  the  analytical 
mode  performed  better  in  the  intuitive  than  the  quasi -rational  rncde. 

Components  of  achievement.  As  the  Lens  Model  Equation  indicates, 
achievement  (r  )  is  a  function  of  knowledge  (G)  and  cognitive  control  ( R  ) . 
Since  two  subjects  might  have  equal  achievement  for  different  reasons --that 
is,  their  knowledge  and  their  cognitive  control  over  the  application  of  their 
knowledge  might  vary  in  a  compensatory  manner--we  determined  the  relative 
contribution  of  these  two  components  for  each  engineer  in  each  cognitive  mode. 
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Table  5 

Number  of  Engineers  with  Highest  Achievement  (r  )  In  Each  Condition 

d 


1st 

Order 

4 

I  >  Q  >  A 

2 

I  >  A  >  Q 

2 

8 

Q  >  I.  >  A 

6 

0  >  A  >  I 

2 

9 

A  >  I  >  Q 

9 

A  >  Q  >  I 

0 

21 

21 

X2  *  2.00,  p  > 

.20  X2  =  15.86, 

p  <  .01 

For  the  intuitive  and  quasi-rational  data  sets,  the  values  of  the 
parameters  of  the  Lens  Model  Equation  were  calculated  from  the  best  fit  linear 
models  of  the  environment  and  of  each  engineer's  judgments.  For  the 
analytical  data  set,  the  best  fit  linear  model  was  used  for  the  environmental 
system,  but  the  engineer's  judgment  policy  was  represented  by  the  formula  he 
produced.  In  some  cases  the  formulas  had  nonlinear  features.  An  additional 
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analysis  was  carried  out  using  a  linear  best  fit  model  of  the  answers 
generated  form  the  engineer's  formula;  the  results  differed  only  slightly 
from  those  produced  by  the  first  procedure  and  will  not  be  reported  here. 

Knowledge.  Table  6  shows  that  the  knowledge  component,  G,  for  thirteen 
of  the  twenty -one  engineers  is  higher  in  the  intuitive  mode  than  in  the 
quasi-rational  and  analytical  modes  (X  =  8.86;  p  <  .02).  Examination  of 
order  also  shows  a  distribution  significantly  different  from  chance  (p  <  .01). 
No  engineer  has  poorest  knowledge  in  the  intuitive  mode.  Thus  we  conclude 
that  if  the  attenuation  of  r  by  the  inconsistency  in  the  task  and  in  each 

Q 

subject's  performance  were  eliminated,  most  engineers  would  have  achieved 
higher  accuracy  of  prediction  in  the  intuitive  mode  than  any  other.  Most 
engineers  would  have  judged  least  accurately  in  the  analytical  mode.  The 
number  of  engineers  whose  quasi-rational  judgments  were  best  would  have  been 
about  the  same  as  the  number  whose  quasi-rational  judgments  were  poorest. 

Cognitive  control.  The  cognitive  control  component,  R$,  is  of  ^course 
significantly  higher  (p  <  .001)  in  the  analytical  mode  than  in  the  other  modes 
because  the  answers  in  the  analytical  mode  were  mechanically  generated  from 
the  engineers'  formulas.  Far  more  engineers  had  a  larger  R$  in  the 
quasi-rational  mode  than  in  the  intuitive  mode  (16  of  21;  p  <  .05).  The 
relation  between  cognitive  control  and  repeated  trials  reliability  is 
discussed  in  Appendix  E. 
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Table  6 


Number  of  Engineers  with  Highest  Knowledge  (G)  in  Each  Condition 


Order 


I  >  0  >  A 


I  >  A  >  Q 


Q  >  I  >  A 
Q  >  A  >  I 


A  >  I  >  Q 
A  >  Q  >  I 


X-  =  8.86,  p  <  .02 


=  18.14,  p  <  .01 


Summary  of  findings  regarding  achievement.  Fewest  engineers  were  most 
accurate  (rfl)  in  the  intuitive  mode,  although  this  finding  is  not 
statistically  significant.  Comparison  of  knowledge  as  measured  by  G,  however, 
indicates  that  even  though  the  engineers  explicated  their  knowledge  and 
applied  it  rigorously,  i.e.,  with  perfect  consistency,  in  the  analytical  mode, 
their  achievement  would  have  been  higher  in  the  intuitive  and  quasi-rational 
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modes  if  they  had  been  equally  consistent  there.  This  result  supports  the 
conclusion  that  engineers  possess  and  can  apply  implicit  knowledge  about 
highway  safety  that  they  do  not  accurately  express  in  rigorous,  retraceable, 
quantitative  form. 

Differential  Pi stributions  of  Errors  under  Different  Cognitive  Modes 

Investigating  the  covariation  between  judgment  and  criterion  does  not 
exhaust  the  possible  methods  for  evaluation  of  performance.  Brunswik 
suggested  (1948,  1956,  pp.  89-91;  see  also  Hammond,  Note  1)  that  the  use  of 
different  modes  of  cognition  may  lead  to  similar  achievement  but  to  very 
different  types  of  errors.  Based  on  the  distinction  between  intuition  and 
analysis  described  by  Brunswik  and  Hammond,  we  test  the  following  hypotheses: 

1.  Errors  made  in  the  intuitive  mode  are  normally  distributed  and  also 
less  frequently  exactly  correct  and  less  frequently  widely  incorrect 
than  errors  made  in  the  analytical  mode; 

2.  The  quasi-rational  mode  produces  an  error  distribution  that  lies 

between  the  distribution  of  errors  in  intuitive  and  analytical 

cognition. 

Rescal ing  procedure.  The  error  scores  were  produced  by  subtracting  the 
accident  rate  for  a  highway  from  the  engineer's  judgment.  The  response  scale 
used  in  the  intuitive  mode  ran  from  1  (=  unsafe)  to  10  (=  safe),  whereas  the 

scales  in  the  quasi-rational  and  analytical  mode  ran  from  0  to  32  accidents 

per  million  vehicle  miles  traveled.  To  make  the  error  distributions 
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comparable,  judgments  made  in  the  intuitive  mode  were  rescaled  onto  the  0  to 
32  scale.  Further,  some  engineers'  analytical  formulas  did  not  produce 
numbers  between  U  and  32.  Their  responses  were  rescaled  by  mapping  the  number 
they  intended  their  formula  to  produce  for  the  safest  road  onto  zero,  and  the 
number  they  intended  for  the  most  dangerous  road  onto  32. 

Hypothesis  1_:  The  error  distributions  in  the  analytical  condition  should 
deviate  from  normal  more  often  than  they  do  in  the  intuitive  condition. 

Procedure.  For  each  engineer  and  for  each  mode,  the  mean  and  standard 
deviation  of  each  engineer's  error  distribution  were  used  to  produce  a  normal 
distribution.  The  null  hypothesis  is  that  the  observed  distribution  of  an 
engineer's  errors  does  not  differ  significantly  from  the  constructed  normal 
distribution.  Therefore,  the  normal  distribution  was  divided  into  six 
categories  expected  to  have  equal  numbers  of  observations  (the  boundaries  were 
located  at  -.97,  -.43,  0,  +.43  and  +.97  standard  deviations  away  from  the 
mean),  and  the  number  of  the  errors  that  fell  into  each  category  was  observed. 

Results.  In  the  intuitive,  quasi-rational  and  analytical  modes  there 
were  few  engineers  (two,  five,  and  three  out  of  twenty-one,  respectively) 
whose  distributions  of  deviations  differed  significantly  f.'om  normal. 
Addi tional ly,  a  comparison  of  the  chi-squares  for  the  deviations  of  these 
error  distributions  from  normal  revealed  that  no  mode  tended  to  have  larger 
deviations  than  the  other  modes.  These  results  offer  no  support  for  the 
hypothesis  that  the  analytical  mode  would  produce  more  non-normal 
distributions  than  the  intuitive  mode. 
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Hypothesis  1  also  asserts  that  more  of  the  engineers'  analytical 
judgments  than  intuitive  judgments  should  have  error  distributions  with 
positive  kurtosis;  the  distributions  of  errors  from  quasi-rational  judgments 
should  lie  between  those  of  the  other  modes. 

Procedure.  Since  a  positive  kurtosis  indicates  a  peaked  frequency 
distribution,  i.e.,  one  in  which  the  answers  are  often  nearly  correct  yet 
occasionally  very  far  off,  we  examined  the  degree  of  kurtosis  in  each 
engineer's  error  distribution. 

Results.  The  results  in  Table  7  indicate  that  the  error  distribution  for 
the  analytical  mode  was  more  peaked  than  for  the  intuitive  inode  for  seventeen 
of  the  21  engineers  (p  <  „U1).  The  quasi-rational  error  distribution  was  more 
peaked  than  the  intuitive  error  distribution  for  seventeen  engineers 
(p  <  .01).  However,  for  only  ten  of  the  engineers  was  the  analytical  error 
distribution  more  peaked  than  the  quasi-rational.  The  results  from 
examination  cf  kurtosis  thus  support  t.he  hypothesis  that  more  engineers  would 
produce  answers  that  were  less  frequently  exactly  correct  in  the  intuitive 
mode  than  in  the  analytical  mode.  Finding  that  the  quasi-rational  error 
distribution  was  not  different  from  the  analytical  was  unexpected. 

A  second  way  of  testing  the  hypothesis  concerning  the  shape  of  error 
distributions  in  each  cognitive  mode  is  to  look  at  each  engineer's  performance 
as  a  whole  rather  than  at  each  answer  individually.  If  the  formula  is 
correct,  achievement  (r  ,  the  correlation  between  answers  and  accident  rate) 

a 

should  be  relatively  high  because  the  formula  should  generate  more  answers 
that  are  nearly  correct  than  intuitive  judgments  do.  If  an  engineer  makes  an 
error  in  a  formula,  al 1  answers  should  be  wrong  and  rfl  should  be  relatively 
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Table  7 

Number  of  Engineers  with  Various  Pairwise  Orders  of 
Kurtosis  of  Error  Distribution 


In  Table  8  the  range  of  correlations  (rfl)  between  answers  and  accident 
rates  for  the  five  best  performances  in  the  analytical  mode  (.643  to  .731)  and 
for  the  five  best  performances  in  the  intuitive  mode  (.618  to  .636)  show  no 
overlap.  From  this  standpoint  the  best  analytical  performances  are  clearly 
superior  to  the  best  intuitive  performances.  However,  the  median  achievement 
in  the  analytical  mode  was  .467,  far  worse  than  the  median  achievement  in  the 
intuitive  mode  (.576);  and  the  analytical  achievements  covered  a  total  range 
of  .957  as  compared  to  the  intuitive  range  of  .565.  Engineers'  performance 
data  as  well  as  analysis  of  error  on  individual  judgments  thus  supports  the 
hypothesis  that  analytical  cognition  is  more  often  very  precise  yet  more  often 
widely  in  error. 
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Table  8 

Achievement  In  Each  Mode 


Intui ti ve 

Quasi-rational 

Analytical 

Span  of  Best  Five 

Engineers'  Achievements 

.618  -  .636 

.624  -  .738 

.643  -  .731 

Median  Achievement 
(all  21  engineers) 

.576 

.516 

.467 

Range  of  Achievement 
(all  21  engineers) 

.565 

.738 

.957 

Experimenter's  Observations  Regardi ng  "Widely  Incorrect"  Answers. 

The  above  analyses  describe  the  distributions  of  errors  produced  by 
different  modes  of  cognition;  but  the  "catastrophic"  errors  that  Brunswik 
predicted  would  occur  in  the  analytical  mode  are  best  illustrated  by  examples 
of  errors  made  by  three  engineers  in  the  analytical  mode: 

1.  Engineer  #2  made  a  careless  arithmetic  error  in  producing  weights  in 
his  formula.  He  first  assigned  a  weight  of  .10  to  each  of  the  ten  cues.  Next 
he  adjusted  the  weights  of  important  cues  to  .12.  Finally,  intending  to 
assign  weights  of  .08  to  cues  he  felt  were  slightly  less  important,  he  wrote 
instead  .8.  Thus  he  gave  the  highest  weight  to  the  cues  to  which  he  wished  to 
give  least  weight.  The  effects  of  his  error  were  "catastrophic":  his 
achievement  (r  )  was  .071,  and  his  mean  error  was  44.176  on  a  scale  he 

a 

intended  to  go  from  0  to  32. 


Direct  Comparison 

Hammond,  Hamm,  Grassia,  and  Pearson 


Page  35 


2.  Although  provided  with  the  information  that  certain  cues  could  assume 
a  value  of  zero,  Engineer  #4  produced  a  formula  in  which  some  of  these  cues 
appeared  in  the  denominators  of  fractions.  As  a  result,  his  answer  was 
indeterminate  on  5  of  the  39  highways. 

3.  Engineer  #10  produced  a  linear  weighted  average  formula,  in  which  he 
inadvertently  used  the  wrong  sign  on  two  cues.  His  achievement  was 
consequently  a  very  low  -.048.  Had  he  not  made  this  error  his  achievement 
would  have  been  .293. 

Since  the  processes  of  intuitive  and  analytical  cognition  cannot  be 
directly  observed,  we  cannot  explain  occasional  poor  performance  in  the 
film-strip  and  bar-graph  tasks  by  citing  the  kind  of  careless,  one-time  errors 
in  execution  of  a  complex  symbolic  procedure  as  we  did  above.  It  is  unlikely 
that  this  kind  of  error  is  occurring  in  the  intuitive  and  quasi -rational 
modes,  however.  Even  if,  in  the  film-strip  and  bar-graph  tasks,  the  engineers 
were  actually  engaging  in  a  step-by-step  cognitive  procedure  similar  to  that 
observed  in  the  analytical  mode,  they  would  have  to  make  the  same  error  during 
each  trial  in  the  task.  Although  a  careless  error  is  guaranteed  to  occur  with 
complete  consistency  when  a  faulty  formula  is  applied  to  thirty-nine  highways, 
it  is  not  guaranteed  by  the  conditions  of  intuitive  and  quasi-rational 
cognition.  On  the  contrary,  since  no  engineer  made  perfectly  consistent 
judgments  in  either  the  intuitive  or  the  quasi -rational  modes,  none  could  be 
expected  to  make  consistent  errors.  Poor  performance  in  the  intuitive  and 
quasi-rational  modes  must  instead  be  attributed  either  to  an  engineer's  lack 
of  knowledge  or  to  his  inability  to  bring  his  knowledge  accurately  to  bear  on 
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Summary  regarding  differential  errors.  When  the  distribution  of  errors 
was  examined,  the  results  provided  no  support  for  the  general  hypothesis  that 
errors  would  be  found  to  be  normally  distributed  in  the  intuitive  mode  more 
often  than  in  the  analytical  mode,  with  errors  in  the  quasi -rational  mode 
intermediate.  However,  the  predicted  differences  in  kurtosis  were  observed: 
engineer's  performances  in  the  analytical  mode  were  more  often  more  accurate 
than  their  performances  in  the  intuitive  mode,  but  occasional  large  errors 
were  produced  in  the  analytical  mode.  Results  in  the  quasi-rational  mode  were 
little  different  from  the  analytical  mode  in  this  regard.  Inspection  of  the 
engineers'  analytical  work  revealed  instances  of  large  errors  as  anticipated 
(misplaced  decimals,  division  by  zero,  and  reversed  signs).  No  similar  errors 
were  found  in  the  other  modes  of  cognition. 

One  reason  that  the  anticipated  differences  in  the  degree  of  normality  of 
error  distributions  were  not  found  nicy  be  that  the  formulas  produced  by  the 
engineers  were  quasi-rational  in  form;  that  is,  many  engineers  developed 
formulas  that  were  analogous  to  a  robust  weighted  average  (eleven  by  a  strict 
criterion,  twenty  by  a  loose  criterion;  see  Appendix  A).  Formulas  of  this 
type  (when  applied  without  errors  in  calculations)  are  as  unlikely  to  produce 
large  errors  as  the  weighted  averages  that  the  evidence  suggests  "he  engineers 
were  using  in  the  intuitive  and  quasi-rational  modes. 
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Confidence 

Ansv.er  Confidence  Compared  wi th  Method  Confidence 

Cognitive  continuum  theory  predicts  that  subjects  will  be  more  confident 
in  answers  produced  by  intuitive  cognition  than  in  answers  produced  by 
analytical  cognition.  Conversely,  the  theory  predicts  more  confidence  in  the 
method  of  analysis  than  in  the  method  of  intuition.  These  predictions  may  be 
tested  by  analyzing  (a)  the  order  of  engineers'  confidence  among  modes  of 
cognition,  for  answer  and  method  confidence  separately,  and  (b)  the  order  of 
engineers'  answer  confidence  and  method  confidence  within  each  mode.  These 
two  approaches  are  diagrammed  in  Figures  3  and  4. 


Intuitive  Quasi-rational  Analytic 


Answer  Confidence 

Method  Confidence 

Mote:  The  direction  of  the  arrows  indicates  decreasing  confidence. 


Figure  3.  Predicted  Relative  Confidence  for  Each  Pair  of  Cognitive  Modes 
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Figure  3  Illustrates  the  expected  order  of  answer  confidence  and  of 
method  confidence  among  modes  of  cognition.  Answer  confidence  should  be 
relatively  high  in  the  intuitive  mode  and  decrease  as  cognition  becomes  more 
analytical.  The  reverse  should  be  true  for  method  confidence. 

Figure  4  represents  the  expected  wi thin-mode  order  of  engineers'  answer 
confidence  and  method  confidence.  In  each  pair  of  modes  engineers  should 
express  more  method  confidence  than  answer  confidence  for  the  mode  closer  to 
the  analytical  pole  on  the  cognitive  continuum. 


IQ  I  A  Q  A 

Answer  Confidence 

Method  Confidence 


Note:  The  direction  of  the  arrows  indicates  decreasing  confidence. 


Figure  4.  Predicted  Relationship  Between  Answer  Confidence  and  Method 
Confidence  for  Each  Pair  of  Cognitive  Modes 


Procedure. 

All  confidence  ratings  in  this  stutty  were  indicated  on  a  1-to-iQ  scale 
where  10  =  high  confidence.  In  the  intuitive  and  quasi-rational  modes,  answer 
confidence  is  indicated  for  each  engineer  by  the  mean  of  the  confidence 
ratings  he  made  for  each  of  his  judgments.  Answer  confidence  in  the 


V# 
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n# 


analytical  mode  and  method  confidence  for  all  three  modes  are  represented  by 
the  means  of  responses  to  particular  conlldence  questions  (see  Appendix  F) 
which  the  engineer  answered  after  making  judgments  or  constructing  a  formula. 

Results 

Orders  of  Answer  and  Method  Confidence.  The  rank  order  of  engineers' 
confidence  across  modes  is  shown  in  Table  9.  For  answer  confidence  the  modal 
order  (10  engineers)  was  exactly  the  predicted  order  (see  Figure  3):  more 
subjects  indicate  more  answer  confidence  in  the  intuitive  mode  than  in  the 
quasi-rational  mode,  and  more  in  the  quasi-rational  than  in  the  analytical 
mode.  As  predicted,  twenty  engineers  had  more  answer  confidence  in  the 
intuitive  than  in  the  quasi-rational  mode  (p  <  .001);  and  sixteen  had  more  in 
the  intuitive  than  in  the  analytical  mode  (p  <  .05);  however,  only  eleven  had 
more  answer  confidence  in  the  quasi-rational  than  in  the  analytical  mode  (NS). 

For  method  confidence,  the  modal  confidence  order  was  I  >  A  >  Q,  which 
was  not  predicted.  Thirteen  and  a  half  engineers  (ties  counted  as  1/2  in  each 
of  the  indicated  orders)  had  more  method  confidence  in  the  analytical  mode 
than  in  the  quasi-rational  mode  (NS,  p  <  .20);  but  only  three  had  more  method 
confidence  in  the  analytical  than  the  intuitive  mode  (p  <  .01  in  the  wrong 
direction),  and  only  3.5  had  more  method  confidence  in  the  quasi-rational  than 
in  the  intuitive  mode  (p  <  .01  in  the  wrong  direction). 
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Table  9 


Count  of  Engineers  Who  Had  Each  Possible  Confidence 
Order  among  the  Three  Modes 


I  >  Q  >  A 
I  >  A  >  Q 

g  >  i  >  a 
g  >  a  >  i 
a  >  i  >  g 
a  >  g  >  i 


X2  =  22.14,  p  <  .001  X2  =  25.78,  p  <  .001 

*  When  an  engineer  had  equal  confidence  In  two  nodes,  .5  count 
was  assigned  to  each  of  the  two  orders.  For  method  confidence, 
there  were  two  engineers  who  had  ties.  Method  Confidence  data 
was  missing  for  one  engineer. 


Answer  Confidence 

i 

Method  Confidence 

10 

3 

5 

11.5* 

1 

2.5* 

0 

1 

6 

2 

0 

0 

Method  Confidence  Compared  with  Answer  Confidence 

Comparison  of  each  engineer's  method  confidence  with  his  answer 
confidence  shows  that  the  majority  of  engineers  had  greater  confidence  in 
answer  than  in  method  for  the  intuitive  and  quasi -rational  modes  and  greater 
confidence  in  method  than  in  answer  for  the  analytical  mode  (see  Table  10). 
Chi-squared  tests  showed,  however,  that  this  finding  is  not  significant. 
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Table  10 

Relative  Answer  and  Method  Confidence  for  Each  Cognitive  Mode* 


I 

Q 

A 

Answer  Confidence  is  greater 

12 

15 

7 

Equal 

2 

Method  Confidence  is  greater 

8 

_ 

6 

12 

*Due  to  missing  data,  N  =  20  for  the  Intuitive  task. 


In  summary,  the  prediction  that  the  engineers  would  have  more  confidence 
in  their  answers  In  the  intuitive  mode  was  born  out,  but  the  prediction  that 
the  engineers  would  have  more  confidence  in  their  method  in  the  analytical 
mode  was  contradicted.  In  the  comparison  of  answer  and  method  confidence 
within  mode,  the  predicted  pattern  of  greater  method  confidence  than  answer 
confidence  in  the  analytical  mode  and  greater  answer  confidence  in  the 
intuitive  irode  was  observed,  but  did  not  »each  statistical  significance.  Most 
engineers  had  greater  confidence  in  the  intuitive  mode  for  both  answers  and 
method.  Judging  safety  was  less  susceptible  to  analytical  cognition  tivn  we 
had  anticipated. 
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roDrlate  Placement  of  Confidence 


Appropriate  placement  of  confidence  can  be  evaluated  in  terms  of  whether 
experts'  confidence  In  their  judgments  reflects  their  achievement--that  is, 
are  they  more  confident  In  the  mode  In  which  their  achievement  is  higher? 

Table  11  displays  for  both  answer  confidence  and  method  confidence  the 
relation  between  relative  confidence  and  relative  achievement  for  each  pair  of 
modes  (Intuitive  versus  quasl-ratlonal ,  Intuitive  versus  analytical,  and 
quasi -rational  versus  analytical). 


In  each  of  the  four-celled  blocks,  the  upper  left  and  lower  right  cells 

contain  the  number  of  engineers  whose  confidence  was  appropriate  given  their 

■  \ 

achievement;  the  upper  right  and  lower  left  cells,  engineers  whose  confidence 
was  misplaced.  For  example,  consider  the  block  relating  relative  answer 
confidence  to  relative  achievement  for  the  intuitive  and  quasl-rational  modes. 
Of  the  thirteen  engineers  who  had  higher  intuitive  achievement  than 
quasl-ratlonal  achievement,  twelve  of  them  had  appropriate  answer  confidence; 
that  is,  they  had  higher  answer  confidence  In  the  intuitive  mode  than  in  the 
quasi-rational  mode.  However,  none  of  the  eight  engineers  who  achieved  higher 
in  the  quasi-rational  mode  than  the  intuitive  mode  had  appropriate  confidence. 


Appropriateness  of  Confidence  Rating  to  Achievement 
a.  Answer  Confidence 
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Both  answer  and  method  confidence  revealed  a  pattern  of  overconfidence  In 
the  Intuitive  mode  as  compared  to  the  quasl-ratlonal  mode  (p  <  .001  and 
p  <  .01,  respectively).  Most  engineers  also  had  Inappropriately  high  answer 
confidence  (p  <  .10)  and  method  confidence  (p  <  .01)  in  the  Intuitive  mode 
when  compared  to  the  analytical  mode.  No  pattern  of  overconfidence  was  seen 
between  the  quasl-ratlonal  and  analytical  modes,  although  the  engineers  were 
very  Inaccurate.  In  fact,  a  null  model  of  random  confidence  judgments  was 
rejected  here  because  the  order  of  engineers'  performance  Is  the  opposite  of 
the  order  of  their  confidence  when  the  quasl-ratlonal  and  analytical  modes  are 
compa2ed. 

Since  the  engineers  received  no  feedback  about  their  achievement  (ra), 
the  appropriateness  of  their  answer  confidence  and  method  confidence  must  also 
be  evaluated  in  terms  of  knowledge  (G ) .  The  same  pattern  of  overconfidence  In 
the  Intuitive  mode  that  was  observed  with  achievement  is  seen  with  knowledge 

f  i 

(Table  12). 

S unwary  Regarding  Appropriateness  of  Confidence 

When  appropriateness  of  confidence  Is  evaluated  In  terms  of  empirical 
achievement  (r  )  most  engineers  were  found  to  be  (a)  more  confident  about 

O 

their  answers  and  also  about  the  method  used  to  produce  their  answers  In  the 
intuitive  mode  than  in  the  other  two  modes;  and  (b)  overconfident  in 
intuitive  cognition  and  underconfident  in  analytical  and  quasi-rational 
cognition.  The  general  pattern  of  overconfidence  in  intuitive  cognition  and 
underconfidence  in  analytical  cognition  remains  when  the  engineers'  knowledge 
(G)  is  examined. 


Appropriateness  of  Confidence  Rating  to  Knowledge 


^Method  Confidence  information  was  missing  for  one  engineer;  therefore  N  =  20. 

**0ne  engineer  expressed  equal  confidence  in  both  modes;  therefore  N  =  19. 

***Expected  frequencies  were  calculated  for  each  ceil  by  A  times  B  times  N,  where  A  =  the  proportion  of 
engineers  who  had  higher  achievement  in  the  given  mode,  B  =  the  overall  proportion  of  engineers  who  had 
appropriate  or  inappropriate  confidence,  and  N  =  the  total  number  of  engineers  {excluding  ties). 
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Summary  and  Discussion 

In  an  effort  to  broaden  and  deepen  comparisons  of  the  efficacy  of 
intuitive  and  analytical  cognition  six  criteria  were  specified:  (a)  direct 
comparisons  should  be  made  between  intuitive  and  analytical  cognition;  (b) 
the  subjects  used  in  the  stu<fy  should  be  analytically  competent  by  virtue  of 
their  professional  training;  (c)  the  concepts  of  intuition  and  analysis,  as 
well  as  the  task  properties  that  induce  each,  should  be  given  clear  meaning; 
(d)  substantive  and  methodological  cognition  should  be  included  in  the  same 
study;  (e)  conclusions  regarding  the  relative  efficacy  of  these  modes  of 
cognition  should  be  norm- independent,  and  thus  not  subject  to  disputed 
standards  of  inductive  inference;  and  (f)  conclusions  should  not  be 
restricted  to  problems  involving  the  logic  of  the  standard  probability 
calculus. 

These  criteria  were  met  by  (a)  requiring  each  subject  to  engage  in 
intuitive  and  analytical  cognition  with  regard  to  the  same  problem,  (b) 
employing  professional  highway  engineers  as  subjects,  (c)  specifying  the 
principal  characteristics  of  intuitive  and  analytical  cognition  and  the 
properties  of  the  tasks  that  induce  them,  { d)  distinguishing  between 
methodological  and  substantive  cognition,  (e)  evaluating  performance  in  terms 
of  empirical  achievement,  and  (f)  employing  problems  that  do  not  require  the 
use  of  the  conventional  probability  calculus. 
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The  results  Indicate  that  within  these  broadened  conditions: 

1.  Achievement  in  a  cognitive  mode  is  greatly  influenced  by  the 
degree  of  consistency  present.  The  absolute  consistency  of  analytical 
cognition  affords  it  an  advantage  In  empirical  achievement  over  intuitive 
and  quasl-ratlonal  cognition,  even  when  the  substantive  knowledge  that  is 
employed  analytically  is  relatively  less  correct.  Conversely,  the 
inconsistency  (low  cognitive  control)  of  intuitive  cognition  will  lead  to 
an  underestimate  of  its  empirical  value,  unless  such  inconsistency  is 
identified  and  removed. 

2.  The  knowledge  implicit  in  intuitive  judgments  was  found  to  be 
empirically  superior  to  quantified,  formalized  knowledge.  When 
inconsistency  was  removed  from  the  engineers'  intuitive  and  quasi -rational 
judgments,  intuition  and  quasi  rationality  were  more  efficacious  than 
analysis  for  most  of  the  subjects. 

3.  Analytical  cognition  was  more  often  highly  accurate,  yet  more 
often  very  inaccurate  when  compared  with  intuitive  cognition  for  most  of 
the  subjects.  Large  errors  were  often  made  In  the  analytical  mode,  but 
seldom  in  the  intuitive  or  quasi-rational  modes. 

4.  When  performance  is  evaluated  in  terms  of  achievement,  engineers 
were  most  confident  in  the  intuitive  mode,  not  only  in  their  answers  (as 
anticipated)  but  in  their  method  as  well  (not  anticipated).  However,  this 
greater  confidence  in  the  intuitive  mode  was  not  always  warranted. 
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These  results  demonstrate  that  a  cognitive  continuum  theory  has  positive 
utility:  it  enables  the  findings  of  this  study  concerning  achievement, 
knowledge,  errors,  and  confidence  to  be  largely  predicted  from  the  nature  of 
the  task  properties  employed.  This  conclusion  has  implications  for  cognitive 
research  as  a  whole.  Research  ranging  from  the  intuition-inducing  tasks  of 
judgment  and  decision  making  to  the  analysis-inducing  tasks  of  problem  solving 
may  also  be  understood  to  be  located  along  a  cognitive  continuum. 
Specification  of  task  properties  and  description  of  cognitive  modes  may  well 
explain  a  variety  of  results  in  cognitive  research  and  thus  reduce  the  current 
isolation  of  research  approaches.  It  may  well  turn  out  that,  like  the 
engineers  in  our  study,  we  know  more  about  our  field  than  we  have  previously 
been  able  to  express. 
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APPENDIX  A 

Characteristics  of  Engineers1  FQrmulas  for 
Safety,  Analytical  Mode 

Eleven  engineers  developed  formulas  that  Involved  only  additive 
combinations  of  the  cues  (median  r  =  .467).  Four  of  these  Involved  only 

a 

linear  functions  of  the  cues  (median  r  =  .281).  For  example: 

a 

Accident  Rate  =  12.8(23  -  (SWIDTH  +  LWIDTH ) )/15  + 

12.8{0BSPM  +  CURVEPM  +  IMTPM)/29  + 

6.4(TRAFMIX  +  PCTNPZ)/U5 

One  expressed  nonlinear  functions  with  graphs  (r.  =  .518);  six  used  tables  to 

a 

express  nonlinear  functions  of  cues  (median  r  =  .496). 

Another  engineer  used  only  a  multiplicative  combination  of  variables 
( r,  =  .670): 

d 

Accident  Rate  =  -ZS<FSW  *  FLH  *  F„pz  *  FGR[J  *  FIpM  * 

fopm  *  fasl  *  ftm  *  Fr») 

where  each  F  Is  a  one-dimensional  table  expressing  a  linear  or  nonlinear 
function  of  the  cue. 

The  remaining  nine  engineers  produced  formulas  that  involved  a  hierarchy 
of  organizing  principles  (median  r  =  .436).  For  example,  one  configurally 

a 

combined  a  subset  of  factors  In  a  table  and  then  added  the  result  together 
with  other  factors.  Five  of  these  hierarchical  formulas  used  adding  and 
multiplying  (median  r  =  .436).  For  example: 

d 

Accident  Rate  =  CURVEPM  +  (60  -  AVESL)/10  - 
(LWIDTH  +  SWIDTH  -  23)/2  + 

(TRAFVOL /10000 ) ( I NTPM  +  OBSPM  -  3)  + 

(TRAFMIX  *  GRADE  *  PCTNPZ)/10000 

Three  engineers  used  adding  as  well  as  tables  (median  r  =  .390).  One 
engineer  used  multiplying  and  tables  (r  =  .641). 

a 
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APPENDIX  B 

Maximal  Guidance  Procedure 


This  procedure  was  designed  to  give  the  engineer  a  maximum  of  guidance  in 
constructing  his  formula  for  safety  In  order  to  prevent  the  commission  of 
minor  errors  and  to  ensure  that  his  formula  adhered  to  principles  of 
measurement  theory  with  which  he  might  not  be  familiar.  The  procedure 
consisted  of  the  following  steps: 

—  Specify  the  answer  scale. 

--  Specify  the  scale  for  each  input  dimension  and  its  overall  relation  to 
the  answer  scale,  and  identify  possible  interactions  with  other 
dimensions. 

—  Group  the  input  dimensions  according  to  their  redundancy,  similarity, 
or  mutual  interactions. 

--  Express  the  formula  as  a  hierarchy  of  groups  of  variables. 

--  Determine  what  organizing  principle  should  be  used  at  eacn  level  of 
hierarchy. 

--  Specify  the  function  form  governing  each  dimension's  input  to  its 
organizing  principle. 

--  Combine  all  the  above  information  into  one  formula. 
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The  engineer  was  guided  through  these  steps  by  a  series  of  forms  which 
contained  Instructions  for  the  steps  and  choice  points,  and  on  wnlch 
Intermediate  steps  were  recorded.  Two  examples  follow.  The  engineer  also 
received  detailed  tutorials  about  Interactions  and  organizing  principles  as 
part  of  the  maximal  guidance  procedure. 
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Form  1:  Answer  Dimension  Form 


Name _ Task  __ 

Answer  dimension's  units _ . 

Range  of  possible  answers:  Low _  High 


A  "natural  0"  on  a  scale  means  that  when  it  Is  called  "0"  there  really  is 
NONE  of  the  quality  being  measured.  If  you  had  a  natural  0,  then  it  would 
make  sense  to  say  that  an  "8"  is  twice  as  much  of  a  thing  as  a  "4";  but  if 
the  0  was  arbitrary,  it  wouldn't  have  that  sort  of  meaning. 

For  example,  if  you  have  savings  of  $10,000,  you  have  twice  as  much  money 
as  if  you  had  $5000,  because  10000  is  twice  5000.  Here  the  $0  is  a  natural  0. 
But  32  degrees  F  is  not  twice  as  warm  as  16  degrees  F,  because  the  0  on  the 
temperature  scale  is  picked  arbitrarily.  In  other  words,  it  does  not  have  a 
natural  0. 

Does  the  answer  dimension  have  a  natural  0?  Yes  _ No _ . 

It  is  useful,  when  considering  numbers  that  measure  a  dimension,  to  ask 
whether  the  intervals  between  the  numbers  have  consistent  meaning,  or  whether 
the  numbers  simply  express  order.  For  example,  is  the  difference  between  a  1 
and  a  2  the  same  as  the  difference  between  an  11  and  a  12?  In  the  above 
measures  of  money  or  temperature,  the  intervals  do  have  consistent  meaning. 
However  if  we  were  to  assign  numbers  to  grades  on  a  test,  where  A  =  1,  B  =  2, 

C  *=  3,  D  =  4,  E  =  5,  and  F  =  6,  the  interval  between  2  and  4  would  be 
different  in  meaning  from  the  interval  between  4  and  6.  All  the  numbers 
convey  is  that  A  is  better  than  B,  etc. 

Do  the  intervals  in  the  answer  scale  have  a  consistent  meaning? 

Yes  No 
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Form  3:  Choice  of  Organizing  Principle 


This  form  Is  for  use  In  deciding  what  organizing  principle  to  use  for 
producing  either  the  final  answer  or  an  Intermediate  product  to  be  plugged  in 
at  a  higher  level  In  the  hierarchy. 

Output. 

Is  this  the  top  level,  producing  the  final  answer?  Yes _ No _ . 

If  so,  what  are  the  units  of  the  final  answer? _ . 

What  Is  Its  range?  Low _ High _ . 

If  this  Is  not  the  top  level,  then  the  output  of  this  organizing 
principle  will  be  Input  for  an  organizing  principle  at  a  higher  level. 

What  organizing  principle  is  used  at  the  next  higher  level? _ . 

What  kind  of  Input  does  it  require?  Units _ . 

Range:  Lowest  point _ ,  Highest  point _ ; 

Does  it  need  to  have  a  natural  0? _ . 

Input. 

List  the  input  dimensions: 


Organizing  Pri nciple. 

What  organizing  principle  do  you  want  to  use  here?  (Refer  to  Sheet  2  for 
guidance  in  your  choice,  and  to  the  Forms  2-i  and  2-g  that  you  used  to 
describe  these  dimensions  to  see  what  kinds  of  interaction  they  have  with  each 
other. ) 

Check  one:  Averaging _ .  Multiplying _ .  Table  (Configural) _ . 

Other 
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APPENDIX  C 

Indepenoent  Confirmation  of  the  Quasi-rational 
Properties  of  the  Safety -Judgment  Task 

Evidence  that  the  safety-judgment  task  was  not  fully  susceptible  to 
analytical  treatment  can  be  seen  in  the  following  examples  of  spontaneous 
comments  by  engineers  during  their  attempts  to  construct  a  formula: 

General . 

"You  kind  of  have  to  be  able  to  juggle  them  all  in  your  head,  you 
know,  in  order  to  assign  a  weight  to  them." 

"I  kind  of  understand  what  I  did." 

"This  Is  too  hard!  Can  I  form  a  committee?" 

"I  haven't  the  vaguest  idea  how  this  is  going  to  come  out." 

Subject  attempts  to  include  curves  and  passing  versus  lane  width. 

"On  the  one  hand,  the  lane  width  and  shoulder  width  lends  itself  to 
giving  the  drivers  a  sense  of  security  -pause-  perhaps  a  sense  of  comfort 
on  the  road  -pause-  where  the  geometry  is  -pause-  not  like  that.  Oh  boy, 

I  don't  know  what  to  do  with  those." 
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Subject  attempts  to  use  average  speed  1 imit.  ( 1 ) 

"But  whether  or  not  you  would  have  more  accidents  per  mile  typically 
on  something  that  is  30  miles  an  hour  or  60  miles  an  hour  is  a  little  bit 
ambiguous,  and  I  don't  know  that  I  cculd  really  properly  consider  that. 

My  inclination  is  to  leave  it  out  for  the  time  being.  Either  you  leave  it 
out  or  conversely  assume  that  it  is  the  only  valid  measure  of  safety.  I 
think  for  the  time  being  I'm  going  to  leave  it  out." 

Subject  attempts  to  use  average  speed  limit.  (2) 

"It's  really  difficult  for  me  to  get  a  feel  for  how  that  influences 
highway  safety.  So  rather  than  trying  to  guess,  I  just  left  it  off, 
recognizing  that,  you  know,  in  a  further  generation  or  a  further 
refinement  of  the  formula,  you  might  be  able  to  come  to  some  conclusion 
about  the  effect  and  include  it." 
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APPENDIX  D 

Notes  on  Think  Aloud  Transcripts 


Engineer  #1 


-organizing  principle--has  feel  for  multiplicative  relationship  but  in 
end  he  merely  rank  orders  cues  and  assigns  weights--additive 

-%NPZ--he  finds  this  cue  ambiguous,  but  finally  resolves  it  as 
diagnostic--he  calls  it  "inverted",  "reciprocal"  (implying  its  function 
form},  but  then  in  his  formula  he  uses  "100  -  %NPZ"  (complement)-*he 
does  not  see  this  cue  as  that  important 

-ASL— also  seen  as  ambiguous — leans  toward  using  it  predictively,  but 
then  leaves  it  out  of  the  formula  because  he  is  unsure  how  to  use  it--he 
mentions  that,  with  regard  to  ASL,  what  is  more  dangerous  is  a  high 
differential,  and  a  low  posted  speed  limit  may  increase  this  (some  cars 
obey  the  low  SL  while  other  cars  disregard  what  they  may  see  as 
unnecessarily  low) 

[not  enough  information  for  engineer  to  know  the  characteristics  of  a 
particular  highway  regarding  their  influence  on  ASL;  also,  he  is 
unaware  of  how  to  express  such  a  relationship  in  a  generalized  formula; 
therefore,  cue  is  simply  left  out  of  formula] 

-indicates  possibilities  for  a  more  complex  organizing  principle  (than 
additive),  i.e.,  nonlinear  -  TV2  -  "but  need  data  to  make  that  judgment" 


Engineer  #5  , 

-Engineer  #5  initially  breaks  accidents  down  into  the  subcategories  of 
property  damage,  injury,  and  fatal ity--elements  affect  different  types 
of  accidents;  these  can  combine  differently  to  get  the  same  accident 
rate  per  mile 

-mentions  cues  that  relate  to  each  other  -  "a  lot  of  these  things  are 
redundant",  e.g.,  0PM  and  SW  [cue  interactions] 

others:  %NP2/SPU/TM,  GRD/TM,  CPM/LW  — "factors  can  aggravate  each 
oth2r" 

-mentions  that  cues  may  be  "cumulative"  rather  than  additive,  i.e., 
interactive 

-the  relationship  of  TM  to  accidents  is  geometric  ( logarithmic?)--"0-5%, 
6-10%,  11-20%,  21-35%"  each  expresses  a  range  of  ratings  that  are  "about 
the  same" 
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-he  alms  for  a  norm/baseline  as  he  develops  his  formula  and  attempts  to 
produce  results  within  the  desired  range.  When  the  results  are  slightly 
off,  he  goes  back  to  one  of  his  matrices  and  adjusts  the  "range"  for  SW 
[actually  he  has  adjusted  the  beta  weight  of  this  cue  in  his  equation] 

Interview: 

-how  would  he  use  historical  research  (which  he  mentions  he  would  like  to 
consult  if  he  could) — in  the  past  he  has  "taken  an  average  rate  and 
modified  it  based  on  whether  you  provide  certain  features  or  detract 
from  those  features";  he  would  look  at  how  "features  stack  up  and 
influence  rates" 

-he  indicates  that  he  knows  he  would  use  a  regression  equation  if  he  had 
the  data 

-he  also  mentions  that  if  the  data  were  available,  the  initial  three 
categories  (property  damage,  injury,  fatality)  would  be  kept 
separate — e.g. ,  some  barriers  may  reduce  fatalities  yet  increase 
property  damage.  Therefore,  each  individual  equation  should  be 
different  for  the  three  categories— of  these,  the  property  damage 
formula  is  the  simplest,  but  the  other  two  are  not  as  high(?)  as 
property  damage,  and  may  be  insignificant  on  the  same  scale  (especially 
fatalities),  causing  the  formula  to  be  basically  a  property  damage 
formula  (which,,  again,  is  simpler) 

-he  indicates  a  definite  awareness  of  multiplicative  relationships 

-confidence  in  method--"I  would  hesitate  to  create  an  equation  for  all 
roadway  conditions  because  you  just  have  to  take  a  look  at  where  the 
roadway  is—" 

-capacity  was  easier  for  him— was  well  researched,  time-space 
relationship  rather  than  individual  characteristics  that  could  influence 
(because  of  variance?) 


Engineer  #8 

-also  starts  with  optimum  value  of  each  characteristic 

-%NPZ— "skrewed"  relationship--"can‘t  get  a  handle  on  how  to  get  that 
incorporated  into  this  (needs  to  use  a  matrix?) 

-uses  trial  and  error  to  obtain  the  relationship  (function)  which  "looks" 
right  (intuitive  feel  for  relationship?) 

-converts  all  ranges  to  a  1-10  scale  based  on  an  optimum  which  may  not  be 
an  end  point 

-SPU—  60  set  as  optimum  irregardless  of  other  factors  [no  interactions] 
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-0PM— range  (0-18)  is  too  complicated  to  calibrate  so  "satisfices"  by 
spreading  1-1U  ratings  out  along  the  range;  therefore  some  cue  values 
have  the  same  rating  (arbi trary?)— 1  .e. ,  12  ano  13  both  equal  a  rating 
of  10. 

-uses  a  weighted  average  organizing  principle 

-he  thought  safety  "was  going  to  be  easy  because  it  would  fit  into  one  of 
the  slots  like  I  did  before,  but  it  doesn't."  (He  had  done  both 
aesthetics  and  capacity) — he  discovers  that  he  cannot  just  subtract  off 
the  number  of  cars  for  each  limiting  factor  like  in  capacity 

-he  observes  that  this  is  like  most  any  formula--"it  gives  you  some  kind 
of  relativity  of  the  things." 

-with  regard  to  the  memory  task,  he  says  there  is  "no  way  in  the  world" 
he  could  remember  his  weighting  system  (how  analytic  was  he?  or  does 
this  generalization  hold?) 


Engineer  #10 

-prioritizes  cues  first  and  then  arranges  them  in  a  linear  additive 
manner,  but  not  weighted  average,  -originally  ranks  cues  by  class 
(cross  section,  geometry,  traffic)  and  then  give  subrankings,  but  then 
in  the  weighting  process  each  cue  is  examined  independently  and  the 
overall  rankings  are  altered  [no  cue  interactions] 

-realizes  he  has  to  normalize  cue  ranges,  i.e.,  give  ratings  to  each 
possible  cue  value  on  a  scale  from  1-10 

-he  forgets  that  his  answer  should  fall  between  0-32 — initially  comes  up 
with  a  range  in  the  hundreds,  but  he  is  unsure  of  the  endpoints— he  says 
one  could  work  them  out 

-After  trying  out  his  formula  on  three  examples,  he  realized  that  he  had 
framed  the  wrong  relationships,  i.e,  most  of  the  cues  had  a  negative 
relation  (correlation)  with  safety  rather  than  a  positive  one.  He 
compensated  for  this  by  switching  the  scale  to  low=safe/high=unsafe  and 
then  readjusting  the  two  positively  related  cues  to  reflect  this  shift. 
All  of  his  cues  were  used  predictively  except  for  %NPZ,  which  he  regards 
as  negatively  related  to  safety  [and  could  be  diagnostic  or 
predictive— 5»NPZ  could  indicate  an  unsafe  road  or  it  could  frustrate 
drivers  and  cause  more  accidents  (both  negativeTJ 

-why  low  confidence?  (5)--"not  enough  thought"  put  into  it— would  like 
to  test  over  a  range  of  highways  "knowing  the  data". 

-memory  task— He  has  problems  remembering  all  the  cues,  much  less  the 
numbers  he  assigned... 
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Engineer  #12 

-begins  to  combine  cues  even  as  he  reads  their  definitions  and  ranges, 
e.g.,  fcNPZ/TV/TM 

•approaches  safety  rating  from  a  "relative"  standpoint  since  the  chances 
of  something  being  zero  [are]  slim"  [answer  for  safety  is  on  a  relative 
scale  though  cues  are  presented  with  their  absolute  ranges  for  tKe 
sample-example  of  prior  knowledge  interf erring  with  an  analytic 
approach?] 

-changes  cues  to  relative  scales  alr>o--i.e.,  (LW+SW)  ranges  from  -2  to  +2 

-wants  an  interaction  between  %H?2  and  TM  but  their  multiplication  can 
produce  a  zero  if  one  term  is  a  zero 

wants  to  say,  that  if  both  %NPZ  and  TM  are  high,  there  is  an  Interaction 
[nonlinear] 

-mentions  "duplication"  (compare  Engineer  #5's  complaint  about 
"redundancy")— have  no  passing  on  steep  grades  and  curves,  "so  it's  all 
kind  of  melded  in  together  and  somehow  we're  duplicating  each  other" 

-"intuitively"  knows  the  kind  of  relationships  he  wants  to  express,  i.e., 
what  kind  of  an  effect  a  certain  cue  interaction  should  have  on  the 
overall  safety  rating,  but  cannot  express  it—goes  through  trial  and 
error 

-he  winds  up  with  a  variable  in  the  denominator  that  could  be  zero— he 
responds  by  saying  "I  won't  allow  it  to  be  zero". 

-he  constantly  tries  out  the  formula  he  has  so  far— to  make  sure  it 
sounds  right  and  nothing  is  too  far  off  (dealing  with  more  concrete 
examples) 

-he  wants  his  worse  case  to  be  32,  so  in  weighting  he  thinks  in 
approximate  percentages  of  32 

-SPDL— diagnostic— "the  lower  the  speed  limit  is,  the  worse  the  situation 
is." 

-ultimately,  his  strategy  is  to  weight  each  "group",  i.e.,  he  produces  a 
"traffic  number",  by  normalizing  the  ranges  of  involved  cues  so  that 
they  add  up  to  a  percentage  of  32. 

[this  comes  out  linear  in  spite  of  his  earlier  expression  as  to  the 
nonl  inearity— he  tries  to  show  some  relationship  by  grouping  in  brackets 
and  multiplying  or  diving  by  a  common  factor] 
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-SPDL--not  weighted  as  heavily  because  "In  setting  the  speed  limit,  It 
considered  all  of  the  other  things  that  were  taken  Into  consideration" 
[diagnostic;  but  he  does  not  see  It  as  an  Interaction  here  between  SPDL 
and  other  cues] 

-during  the  discussion  he  expresses  an  Interaction  between  LW  and  0PM 
that  he  did  not  Include  in  his  formula 

-some  cues  can  be  plugged  directly  into  his  formula  (e.g.,  IPM),  while 
others  (e.g.,  SW )  have  been  converted  to  a  different  scale--negat1ve, 
neutral,  or  positive  [SW,  as  he  expresses  it  verbally,  may  be 
third-order  relationship] 


Engineer  #15 

-he  recognizes  that  the  safety  rating  must  be  in  terms  of  million  vehicle 
miles  travelled  and  so  TV  will  be  a  component  of  the  rating  rather  than 
strictly  a  factor 

-wants  to  develop  weights  that  would  predict  the  number  of  accidents 
which  would  then  be  correlated  with  the  required  tange  (0-32)  [he  asks 
"can  I  do  that?"] 

-uses  TV  on  both  sides  of  his  equation  (now  using  as  a  factor) 

-he  sees  the  complexity  of  the  task  but  attempts  to  "keep  it  simple" 

-weighting  factors  were  the  "factors  of  Importance",  which  then  had  to  be 
put  in  t.he  same  "context"  (normalized)  by  giving  them  a  safety 
rating--produced  a  "relative  safety  value"  (actually  he  just  changed  the 
range  on  each  cue  to  1-5)  but  relationships  are  not  quite  linear 

-%NPZ--diagnostic--the  higher  the  percentage,  the  worse  the  condition 

-SPD— predictive--the  lower  the  speed  limit,  the  safer 

-multiplies  ranges  by  weighting  factors  to  amplify  them 

-assumes  a  linear  relationship  between  cues  because  he  has  "built  any 
curve  aspects  into  the  judgments  here"  (the  cue-criterion  relationships) 

-he  comments  that  this  empirical  formula  is  probably  too 
complicated--after  trying  it  out,  one  would  probably  find  that  he  could 
get  just  as  good  results  by  dealing  with  four  or  five  factors  rather 
than  all  of  the  dimensions.  He  nevertheless  used  all  of  them  in  his 
formula  "because  there  was  a  judgment  built  into  this  whole  thing  In  the 
first  place  that  these  factors  were  important." 
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-places  TV  In  the  denominator  to  give  "a  means  of  comparing  different 
length  segments"  and  "differing  volumes "--what  he  doesn't  realize  is 
•  that  the  rating  is  already  in  terms  of  million  vehicle  miles  travelled 

(or  maybe  he  just  forgets  this) 

-in  going  over  his  steps  he  recalls  wanting  to  "use  a  system  where  I  come 
up  with  an  intuitive  judgment  about  the  safety  of  the  thing  as  judged  by 
the  dimensions  given"  [underlining  ours] 

-"If  you're  going  to  have  a  practical  formula,  you've  got  to  have  a  way 
where  people  can  use  it  quickly."  [In  the  process  of  developing  a 
"practical  formula"  that  can  be  applied  easily  does  one  oversimplify 
complex  reiationships--thereby  reducing  validity?  (and  accuracy?)] 
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APPENDIX  E 

Comparison  of  Repeated  Trials  Reliability  with  Cognitive  Control 

Fourteen  of  the  engineers  had  higher  Rc's  than  Rs's  in  the  intuitive 
inode.  This  finding  suggests  that  these  engineers  were  using  information  iri  a 
nonlinear  way,  and/or  that  the  pictorial  presentation  contained  information 
not  present  in  the  ten  cues  provided  in  the  bar-graph  and  formula 
presentations.  Both  possibilities  were  investigated,  and  neither  can  be  ruled 
out  with  certainty.  The  evidence  suggests,  however,  that  (a)  the  engineers 
did  not  make  systematic  or  substantial  use  of  Information  in  a  nonlinear 
fashion;  (b)  in  the  film-strip  condition  they  occasionally  used  information 
that  was  not  present  in  the  other  tasks;  (c)  the  engineers'  occasional  use  of 
additional  information  in  the  intuitive  task  does  not  account  for  the  higher 
repeated  trials  reliabilities. 

The  contribution  of  the  nonlinear  component  of  the  Lens  Model  Equation 
was  investigated  to  determine  whether  a  significant  relation  existed  between 
the  portions  of  the  variances  of  the  criterion  and  the  judgments  that  were  not 
explained  by  their  respective  linear  models.  Minimal  evidence  of  such 
nonlinear  knowledge  was  found.  The  mean  absolute  value  of  the  nonlinear 

/  2  f  2 

component  (C  J  1  -  Rg  /  1  -  R$  )  was  .056  in  the  intuitive  mode  and  .048  in 
the  quasi -rational  mode. 
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That  some  engineers  In  the  Intuitive  condition  made  use  of  Information 
present  only  In  the  film  strips,  at  least  on  some  trials,  is  almost  certainly 
true.  For  instance,  residential  driveways  entering  the  highway  were  evident 
and  could  have  been  taken  into  account. 

The  most  plausible  explanation  for  the  higher  repeated  trials  reliability 
in  the  film-strip  condition  than  in  the  bar-graph  condition  is  that  the 
engineers  recognized  the  repeated  film  strips  and  recalled  their  previous 
answers.  There  are  two  reasons  for  accepting  this  explanation:  (a)  it  was 
obvious  to  the  engineers  that  they  were  viewing  a  film  for  a  second  time,  and 
some  of  them  remarked  that  they  remembered  their  previous  answer;  (b)  whereas 
repeated  trials  reliability  was  higher  than  multiple  R  in  the  film-strip 
condition,  it  was  higher  for  only  two  engineers  in  the  bar  graph  condition; 
and  no  engineer  in  this  condition  indicated  that  he  recognized  a  repeated 
trial.  In  short,  the  fact  that  fourteen  of  the  21  engineers  had  higher 
repeated  trials  reliabilities  in  the  film-strip  condition  is  almost  certainly 
the  result  of  an  artifact  of  memory. 
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APPENDIX  F 
Confidence  Questions 

Answer  Confidence.  In  the  analytical  mode  the  engineer  was  asked,  "How 
confident  are  you  that  your  formula  will  work  well  with  any  particular  2-lane 
rural  Colorado  highway?"  once  after  producing  the  formula  and  again  at  the  end 
of  the  session.  The  mean  of  his  responses  is  the  measure  of  his  answer 
confidence  in  the  analytical  mode. 

Method  Confidence.  In  the  intuitive  and  quasi-rational  modes  * ;ie  mean  of 
the  confidence  ratings  on  the  following  three  questions  from  the  Self  Report 
form  is  the  measure  of  method  confidence: 

1.  How  confident  are  you  that  your  method  for  making  these  safety 
judgments  is  correct? 

2.  How  well  does  this  presentation  help  you  make  safety  judgments? 

3.  How  accurate  do  you  think  one  could  be  when  making  these  judgments  in 
this  way? 

The  measure  for  analytical  method  confidence  is  the  mean  of  the  ratings  on  the 
following  questions: 

1.  How  confident  are  you  that  your  formula  is  correct?  (answered  at  two 
different  times) 

2.  How  accurate  do  you  think  a  formula  for  judging  the  capacity  of  highways 
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